Robust laparoscopic instruments tracking using colored strips

To assist surgeons in the acquisition of the required skills for the proper execution of the laparoscopic procedure, surgical simulators are used. During training with simulators it is useful to provide a surgical performance quantitative evaluation. Recent research works showed that such evaluation can be obtained by tracking the laparoscopic instruments, using only the images provided by the laparoscope and without hindering the surgical scene. In this work the state of the art method is improved so that a robust tracking can run even with the noisy background provided by realistic simulators. The method was validated by comparison with the tracking of a â\u80\u9cchess-boardâ\u80\u9d pattern and following tests were performed to check the robustness of the developed algorithm. Despite the noisy environment, the implemented method was found to be able to track the tip of the surgical instrument with a good accuracy compared to the other studies in the literature

ADVANCED STRATEGIES FOR AUGMENTED REALITY SIMULATION-BASED SURGICAL TRAINING

The augmented reality (AR) simulation represents a great potential for learning and transferring new skills, not only in minimally invasive surgery (MIS) but also in open surgery. AR simulators retain the natural haptic feedback of physical simulators and the performance evaluation tools of VR simulators. However, commercial AR simulators (e.g. CAE Healthcare ProMIS) are often based on physical models that are neither highly detailed, nor anthropomorphic and they have costs comparable to those of VR simulators. In addition, available commercial AR systems only provide visual instructions and guidance information to the trainee and do not allow either to update the virtual anatomy following deformations impressed on physical anatomical models, or to add simulated physiological functionalities which increase the realism of the system. This thesis proposes an advanced strategy to develop AR simulation platforms which combine highly detailed physical models and virtual reality information in a surgical scene, and integrates deeply both visual and tactile AR, and acoustic functionalities, improving the state of the art on AR simulators. The strategy can be applied to simulate all surgical procedures involving the task of identification and isolation of generic tubular structures, both in laparoscopic and open approach. The cholecystectomy, one of the most common surgical procedure, has been chosen as benchmark procedure to demonstrate the potentialities of the proposed strategy. Specifically, the simulator is designed for the task of identification and isolation of the Calot's triangle, the most critical phase of cholecystectomy. The proposed simulator includes all the anatomical structures which could be either seen or touched during the execution of the procedure: patient-specific physical replicas of liver, gallbladder, pancreas, abdominal aorta, esophagus-stomach-duodenum and realistic physical models of biliary ducts (BT), arterial tree (AT) and connective tissue. A key feature of AT and BT is the ability to be easily and separately replaced on demand, with different specific anatomical variations, thus realizing training sessions with increasing complexity. All the anatomical structures, except from the connective tissue which has to be dissected, are fabricated using materials which are extremely durable over time and reusable, such as silicone and nitinol tubes, thus reducing training costs. The BT and the AT are sensorized with Electromagnetic (EM) sensors and tracked in real time for the implementation of AR functionalities: tactile AR in open surgery and visual AR both in open and in laparoscopic surgery. The laparoscope is localized in real time, by means of an additional camera for tracking of a structured marker, in order to allow its maneuverability as commonly occur in laparoscopic procedures. The open surgery mode includes tactile AR functionality and a wearable haptic device to provide pulsed feedback during palpation, the latter tracked by means of an EM localization. AR is used as an aid to accurately show the Calot's triangle position, by means of AR visualization mode, in the laparoscopic approach, and to provide pulsed feedback during palpation, by means of AR tactile, in the open approach. This is possible thanks to the versatility of the proposed AR simulator which allows the transition from the laparoscopic to the open approach by replacing the laparoscope with a wearable haptic device. In addition, the simulator includes an electrical apparatus to report surgical errors. An attractive feature of this system is that it allows the user to interact in real-time with virtual models and to perceive the implemented organ’s functionality, blood vessels pulsation, in a natural way. Preliminary tests show that the AR simulator satisfies all the basic specifications: good anatomic appearance, modularity, reusability, cost-effectiveness, robustness, ability to report surgical errors and ability to provide AR aids. General surgeons positively evaluated the realism both the connective tissue and the AT and the BT embedded into the connective tissue. A positive qualitative feedback has been received regarding the usefulness of both acoustic functionality, to signal surgical errors, and of the AR scene, as an aid to detection of AT and BT. In conclusion, based on the studies performed in the context of this thesis and the users’ feedbacks, the AR simulator is likely to be considered as a potential training tool to learn the task of identification and isolation of anatomical tubular structures not clearly visible. In conclusion, as demonstrated in this thesis, a great step forward in surgical simulation will be possible developing hybrid AR simulators with a deep integration between real and virtual components

Simulatore fisico paziente-specifico per colecistectomia laparoscopica

Lo sviluppo delle tecnologie di simulazione è stato fortemente promosso nel campo chirurgico con l’avvento della chirurgia laparoscopica che richiede particolari abilità psicomotorie ed elevati livelli di coordinazione occhio-mano. I simulatori fisici, ottenuti usando modelli in plastica, gomma e lattice inseriti in box, imitano meglio l’interazione in termini di ritorno di forza con l’anatomia rispetto ai simulatori basati sulla realtà virtuale attualmente disponibili. Questi modelli fisici sono usati per rappresentare differenti organi e patologie e permettono di eseguire specifici task come il taglio, sutura, afferraggio o clipping. Un limite dei simulatori fisici attuali è che riproducono una o poche strutture anatomiche aventi anatomia standard. Altro limite è relativo ai task distruttivi che richiedono un nuovo, e a volte costoso, phantom per ogni prova. Infine, i simulatori fisici non prevedono un sistema di valutazione automatica e quantitativa delle prestazioni, ma l’apprendista deve essere valutato da un chirurgo esperto che giudica se l’atto chirurgico è eseguito correttamente. Lo scopo di questo lavoro è la progettazione e realizzazione, a livello prototipale di un simulatore fisico paziente-specifico per colecistectomia laparoscopica che si propone di superare i limiti precedentemente menzionati. In particolare il simulatore è volto a simulare l’atto di identificazione e isolamento delle strutture cistiche che delimitano il triangolo di Calot, fase cruciale dell’intervento. Il simulatore, sviluppato a partire dall’elaborazione di immagini TC anonimizzate, comprende la replica, in termini di topologia, morfologia e colore delle strutture rilevanti per un intervento di colecistectomia: fegato, colecisti, dotto cistico, coledoco, dotto epatico comune, albero arterioso comprendente arteria cistica e arterie epatiche, e piccolo omento. Le immagini preoperatorie TC sono state segmentate usando uno strumento semiautomatico per ottenere modelli 3D di fegato, colecisti, vie biliari e albero arterioso. Partendo da questi modelli 3D, sono stati disegnati gli stampi per la replica delle parti anatomiche che sono stati ottenuti mediante prototipazione rapida. La replica del fegato è stata ottenuta colando silicone platinum-cured e mescolato con pigmenti colorati all’interno del relativo stampo. Il modello cavo della colecisti è stato realizzato applicando vari strati di silicone sullo stampo ottenuto come replica del suo volume interno. Per ottenere l’elasticità ottimale, spessore e robustezza, agli strati di silicone, sono state aggiunte fibre naturali. Infine la colecisti è stata riempita con idrogel. Il piccolo omento è stato prodotto sotto forma di sottili fogli di silicone rinforzato con cotone. Al silicone è stato inoltre aggiunto un additivo siliconico (Slacker) per silicone platinum-cured. Lo Slacker permette di ottenere una consistenza molto simile a quella del tessuto naturale e di ottenere inoltre la sua adesione alle altre parti anatomiche. Una comune strategia è stata usata per la riproduzione del dotto cistico, coledoco e dotto epatico comune (vie biliari). La strategia ha previsto l’uso di maglie schermanti conduttive e flessibili ricoperte con uno strato di silicone. L’albero arterioso è stata realizzato usando un’anima in nitinol, ricoperta con due strati di silicone colorato. Ogni componente del simulatore è stata valutata da tre chirurghi esperti e quattro specializzandi di chirurgia generale. Sulla base dei loro commenti sono state apportate delle modifiche e si è raggiunto il 90% di opinioni positive. Tutte le componenti sono state assemblate all’interno di un torso di un phantom commerciale, ottenendo una loro posizione realistica al fine di realizzare un ambiente verosimile. Infine, la maglia schermante all’interno del dotto biliare e l’anima in nitinol all’interno dell’albero arterioso sono state connesse elettricamente ai poli negativi di due distinti allarmi acustici. Un uncino chirurgico, usato comunemente per questo tipo di intervento, è stato connesso alla massa di un alimentatore il cui polo positivo è stato connesso direttamente agli allarmi. In questo modo, se l’uncino taglia la copertura di silicone dei dotti biliari o dell’albero arterioso, e tocca l’elemento conduttivo interno, per la chiusura del circuito si ha l’attivazione del relativo allarme che segnala l’errore rispettivamente di lesione dei dotti biliari o dell’albero arterioso. Queste strutture sono state incluse nell’omento simulato come in un reale paziente. Per simulare lo pneumoperitoneo caratteristico della procedura laparoscopica è stata realizzata una copertura in materiale plastico termoformabile con finestre in silicone soffice per l’inserimento degli strumenti laparoscopici. L’ultimo passo ha previsto la valutazione del simulatore eseguita da due chirurghi che hanno simulato l’isolamento dell’dotto cistico e dell’arteria cistica. Durante l’isolamento delle strutture, a volte l’omento si è mosso in modo non realistico in quanto lo slacker, usato anche come colla, non ha sempre aderito adeguatamente. I chirurghi hanno valutato positivamente il realismo del manichino ai fini di un corretto apprendimento dell’ uso degli strumenti chirurgici laparoscopici. Inoltre hanno valutato che le parti interne del manichino offrono un feedback di forza e una risposta meccanica al taglio realistici ed hanno apprezzato entusiasticamente l’idea dei contatti elettrici per simulare la lesione del dotto cistico e dell’arteria. Un altro punto di forza di questo simulatore è che al termine di ogni sessione di addestramento solo il piccolo omento dovrà essere sostituito tenendo così bassi i costi per il training. Questo approccio permette la realizzazione di dotti biliari e albero arterioso con differenti morfologia e topologia, offrendo la possibilità all’utente di cimentarsi in situazioni rese complesse da varianti anatomiche. Il prototipo di simulatore realizzato può essere ritenuto un’utile piattaforma di training laparoscopico ed eventualmente robotico per i giovani chirurghi che potranno imparare velocemente ed in modo sicuro la procedura chirurgica.

Augmented Reality, Mixed Reality, and Hybrid Approach in Healthcare Simulation: A Systematic Review

Simulation-based medical training is considered an effective tool to acquire/refine technical skills, mitigating the ethical issues of Halsted’s model. This review aims at evaluating the literature on medical simulation techniques based on augmented reality (AR), mixed reality (MR), and hybrid approaches. The research identified 23 articles that meet the inclusion criteria: 43% combine two approaches (MR and hybrid), 22% combine all three, 26% employ only the hybrid approach, and 9% apply only the MR approach. Among the studies reviewed, 22% use commercial simulators, whereas 78% describe custom-made simulators. Each simulator is classified according to its target clinical application: training of surgical tasks (e.g., specific tasks for training in neurosurgery, abdominal surgery, orthopedic surgery, dental surgery, otorhinolaryngological surgery, or also generic tasks such as palpation) and education in medicine (e.g., anatomy learning). Additionally, the review assesses the complexity, reusability, and realism of the physical replicas, as well as the portability of the simulators. Finally, we describe whether and how the simulators have been validated. The review highlights that most of the studies do not have a significant sample size and that they include only a feasibility assessment and preliminary validation; thus, further research is needed to validate existing simulators and to verify whether improvements in performance on a simulated scenario translate into improved performance on real patients

Patient-specific ultrasound liver phantom: materials and fabrication method

Purpose : An anatomically realistic ultrasound liver phantom with tissue-specific distinct signal properties is needed for training of novices in diagnostic and interventional procedures. The main objective of this work was development and testing of a new durable liver ultrasound training phantom for use with a hybrid simulator. Methods : A liver ultrasound phantom was fabricated in four main phases: materials selection, segmentation of CT images and realization of 3D models, vessel and lesion realization, and final assembly with silicone casting. Silicone was used as basic material due to its durability and stability over time. Several additives were analyzed and mixed with the polymer to reproduce the echogenicity of three simulated soft tissue types: parenchyma, lesions, and veins. Results : Cysts and vessel trees appear anechoic in the B mode ultrasound images when realized with pure silicone. The liver parenchyma, hypoechoic, and hyperechoic lesions were realized with different concentrations of graphite and Vaseline oil to increase their relative echogenicity. These materials were successful for creation of an ultrasound liver phantom containing simulated blood vessels and lesions. Conclusion : The phantom reproduces the human liver morphology and provides vessels and lesions ultrasound images with recognizable differences in echogenicity. The speed of sound in the simulated materials is inaccurate, but the problem can be overcome via software adjustment in a hybrid simulator

Augmented Reality, Mixed Reality, and Hybrid Approach in Healthcare Simulation: A Systematic Review

Simulation-based medical training is considered an effective tool to acquire/refine technical skills, mitigating the ethical issues of Halsted’s model. This review aims at evaluating the literature on medical simulation techniques based on augmented reality (AR), mixed reality (MR), and hybrid approaches. The research identified 23 articles that meet the inclusion criteria: 43% combine two approaches (MR and hybrid), 22% combine all three, 26% employ only the hybrid approach, and 9% apply only the MR approach. Among the studies reviewed, 22% use commercial simulators, whereas 78% describe custom-made simulators. Each simulator is classified according to its target clinical application: training of surgical tasks (e.g., specific tasks for training in neurosurgery, abdominal surgery, orthopedic surgery, dental surgery, otorhinolaryngological surgery, or also generic tasks such as palpation) and education in medicine (e.g., anatomy learning). Additionally, the review assesses the complexity, reusability, and realism of the physical replicas, as well as the portability of the simulators. Finally, we describe whether and how the simulators have been validated. The review highlights that most of the studies do not have a significant sample size and that they include only a feasibility assessment and preliminary validation; thus, further research is needed to validate existing simulators and to verify whether improvements in performance on a simulated scenario translate into improved performance on real patients

Simulatore chirurgico con realtà aumentata, in particolare per strutture anatomiche deformabili

Abstract not availabl

AR visualization of "synthetic Calot's triangle" for training in cholecystectomy

The most challenging step of cholecystectomy is the identification of Calot’s triangle. Adequate training by the surgical operators may reduce intraoperative risks due to anatomical misinterpretation. In this context, augmented reality (AR) can offer an excellent contribution to traditional education, by allowing the implementation of hybrid simulators featuring the visualization of hidden anatomical structures. The aim of this study is to evaluate whether a complete AR visualization of the Calot’s triangle is possible by means of a commercial electromagnetic tracking system and a limited number of sensors. A strategy for the manufacturing and sensorization of the Calot’s triangle, is proposed. Sensorized replica of the most common configurations of the arterial tree and biliary tree were fabricated by using seven electromagnetic coils. Accuracy in AR visualization was preliminarily evaluated in quantitative and qualitative terms. Obtained results confirmed the feasibility of the proposed AR approach to track and visualize the Calot’s triangle. The achieved accuracy is adequate for training purposes and can be employed to aid the trainee in the recognition of the Calot's triangle. Based on these results, the developed prototypes will be integrated into our cholecystectomy simulator for a complete assessment

Augmented reality visualization of deformable tubular structures for surgical simulation

Background Surgical simulation based on augmented reality (AR), mixing the benefits of physical and virtual simulation, represents a step forward in surgical training. However, available systems are unable to update the virtual anatomy following deformations impressed on actual anatomy. Methods A proof-of-concept solution is described providing AR visualization of hidden deformable tubular structures using nitinol tubes sensorized with electromagnetic sensors. This system was tested in vitro on a setup comprised of sensorized cystic, left and right hepatic, and proper hepatic arteries. In the trial session, the surgeon deformed the tubular structures with surgical forceps in 10 positions. Results The mean, standard deviation, and maximum misalignment between virtual and real arteries were 0.35, 0.22, and 0.99 mm, respectively. Conclusion The alignment accuracy obtained demonstrates the feasibility of the approach, which can be adopted in advanced AR simulations, in particular as an aid to the identification and isolation of tubular structures